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Abstract 

This report documents a standard- setting study to determine recommended minimum scores (cut 
scores) needed on the Core and Advanced iSkills™ assessments for examinees to be considered 
at a foundational level of ICT literacy skill. Two foundational levels — one for each iSkills 
assessment — had been specified previously by the National ICT Literacy Policy Council, a group 
that was formed by the National Forum on Information Literacy (http://www.infolit.org) and that 
serves as the certification board for ICT literacy standards. Following a modified, extended 
Angoff method and a counterbalanced design, two independent panels converged on 
recommended scores corresponding to the foundational levels: 165 (on a scale of 0-300) for the 
Core iSkills assessment and 575 (on a scale of 400-700) for the Advanced iSkills assessment. 
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Executive Summary 

This report documents a standard- setting study held July 20-23, 2007, at Educational 
Testing Service (ETS) in Princeton, New Jersey. The purpose of the study was to determine 
recommended minimum scores needed on the Core and Advanced iSkills™ assessments for 
examinees to be considered at a foundational level of information and communication 
technology (ICT) literacy skill. 

Two foundational levels — one each for the Core iSkills assessment and Advanced iSkills 
assessment — had been specified previously by the National ICT Literacy Policy Council, a group 
that was formed by the National Forum on Information Literacy (http://www.infolit.org) and that 
serves as the certification board for ICT literacy standards. The core foundational level describes 
the minimum ICT literacy skills a student should have upon entering the first year of post- 
secondary education. The intermediate foundational level describes the minimum ICT literacy 
skills a student should have upon entering upper-division post- secondary coursework or entering 
the workforce. The label intermediate was used for the Advanced iSkills assessment because the 
council believed that the label was a more appropriate description of a foundational level of skill. 
The two foundational level descriptions were reviewed by a larger community of educators and 
workforce representatives and suggested revisions were accepted by the council. The council- 
approved foundational level descriptions were key components of the standard-setting process 
and are shown in Appendix F. 

iSkills Assessments 

In designing ETS’s iSkills assessment, librarians and information literacy experts 
collaborated to produce a measure of ICT literacy: the information literacy skills that arise in 
technical environments. Both the Core and Advanced iSkills assessments consist of 
approximately 60 items derived from performance on 15 interactive, performance-based tasks. 
The scoring of the items follow rubrics that specify the nature of responses needed to gain full 
credit (1), partial credit (0.5), or no credit (0). The overall raw score on the assessment is the sum 
of all item scores. 
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Standard-Setting Process 

Eighteen educators, each with direct experience with the information literacy of high 
school or college students, were divided into two subpanels. These subpanels independently 
made standard-setting judgments for each item in both tests; the order in which the subpanels 
considered each assessment was counterbalanced. On the final day of the study, the panel 
reconvened to reconcile any differences in their respective recommended cut-score judgments. 

The standard- setting study followed a modified, extended Angoff procedure. Each 
panelist first independently judged the average score that would be received on each of the 60 
items by 100 just qualified candidates (JQCs; examinees just meeting the requirements of the 
foundation level corresponding to the assessment being considered). Members of each subpanel 
discussed their Round 1 judgments, and this process continued for a second round of judgments 
and discussion, followed by a third round of judgments. Item-level data were introduced during 
the Round 1 discussion and classification outcomes (percentage of examinees meeting or 
exceeding the preliminary cut scores) were presented during the Round 2 discussion. The cut 
scores derived from the Round 3 judgments were reconciled in full panel discussions. 

Recommended Cut Scores Corresponding to Foundational Levels 

The reporting scale for the Core iSkills assessment ranges from 0 to 300, calibrated to a 
mean of 150 and standard deviation of 35. The panel recommended a cut score of 165 . 
Considering the first year students who have taken the Core iSkills assessment since 2006, this 
cut score results in approximately 24% of community/technical college first-year students and 
39% of 4-year college freshmen meeting or exceeding the core foundational level of ICT literacy 
skill. The reporting scale for the Advanced iSkills assessment ranges from 400 to 700, calibrated 
to a mean of 550 and standard deviation of 35. The panel recommended a cut score of 575 . 
Considering the juniors and seniors who have taken the Advanced iSkills assessment since 2006, 
this cut score results in approximately 27% of these students meeting or exceeding the 
intermediate foundational level of ICT literacy skill. 

In December 2007, the National ICT Literacy Policy Council approved the recommended 
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Introduction 

This report documents a standard-setting study held July 20-23, 2007, at ETS in 
Princeton, New Jersey. The purpose of the study was to determine recommended minimum 
scores needed on the Core and Advanced iSkills™ assessments for examinees to be considered 
at a foundational level of information and communication technology (ICT) literacy skill. The 
recommended minimum scores (cut scores) will be presented to the National ICT Literacy Policy 
Council — a group that was formed by the National Forum on Information Literacy 
(http://www.infolit.org) and that serves as the certification board for ICT literacy standards — for 
acceptance or adjustment before public release of the cut scores. 

The next sections describe the foundational levels, iSkills assessments, and standard- 
setting process (overall design, panelist descriptions, and judgment process) in more detail. 
Following those sections are the results, including initial evaluations by the panelists, cut-score 
recommendations, and final evaluation of the standard-setting process. 

Foundational Levels 

The National ICT Literacy Policy Council (http://www.infolit.org/policycouncil.doc) met 
on February 5 and 6, 2007, in Washington, DC to draft definitions for expected levels of ICT 
literacy skill performance. These definitions, presented in Appendix F, represent the 
foundational level of ICT literacy skills — what is minimally expected of students leaving either 
secondary education to enter college (core foundational level) or their second year of 
postsecondary education to enter upper-division coursework or the workforce (intermediate 
foundational level). The foundational levels, therefore, describe standards of performance 
expected of students at two transition points in their education. These definitions were 
subsequently reviewed by other representatives (i.e., external to the policy council) of the high 
school, community college, 4-year college/university, and workforce communities before being 
finali z ed and accepted by the policy council. 

In defining these foundational levels, council members considered ICT literacy-related 
standards — such as the Information Literacy Competency Standards for Higher Education 
(Association of Colleges and Research Libraries, 2000) — as well as the ICT Literacy higher 
education framework (http://www.ets.org/iskills), which forms the basis for the ETS iSkills 
assessments. Each foundational level description is intended to help define a minimal cut score 
on either the Core or Advanced iSkills assessment. Each level definition includes a general 
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summary description and then specific indicators addressing the seven content areas comprising 
the ETS ICT literacy framework that underlies the iSkills assessments: define, access, evaluate, 
manage, integrate, create, and communicate (see Table 1 for definitions of these areas). The 
policy council reviewed each definition to assure that the general description and specific 
indicators were aligned (mutually reinforcing) and that there was a clear and appropriate 
progression of skills from the core foundational level to the intermediate foundational level. 

iSkills Assessments 

The Core iSkills assessment measures the ICT literacy skills of students who are making 
the transition from high school to the first year of post-secondary education; these are students 
entering either community college or a 4-year college program. The assessment consists of 63 
items within 15 performance-based tasks. Each item is scored 0, 0.5, or 1, and collectively the 
tasks address the seven content areas of the ICT literacy framework (Table 1). The reporting 
scale ranges from 0 to 300, with the mean set to 150 and the standard deviation set to 35, based 
on a calibration group of test takers who took the assessment in early 2006. 

The Advanced iSkills assessment measures the ICT literacy skills of students who are 
making the transition either from second-year post- secondary education to third-year or the 
workforce. The assessment consists of 58 items within 15 tasks. Each item is scored 0, 0.5, or 
1, and collectively the tasks address the same seven ICT literacy content areas as the Core 
iSkills assessment. The reporting scale ranges from 400 to 700, with the mean set to 550 and 
the standard deviation set to 35, based on a calibration group of test takers who took the 
assessment in early 2006. 

Further details on these assessments and their development are provided on the iSkills 
Web site (http://www.ets.org/iskills) and in Katz (2007). 

Panel Members 

Panelists were 18 educators (secondary and post- secondary levels) who, collectively, are 
familiar with the ICT literacy skills of students at various levels, including high school seniors, 
students entering college, college students moving to upper-level coursework, and community 
college students leaving higher education to enter the workforce. These levels represent the 
target populations for the Core and Advanced iSkills assessments. Panelists were drawn from 
recommendations made by members of the National ICT Literacy Policy Council, the National 
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Table 1 

ETS ICT Literacy Framework 

Define: Understand and articulate the scope of an information problem in order to facilitate the electronic search for information, 
such as by 

• Distinguishing a clear, concise, and topical research question from poorly framed questions, such as ones that are 
overly broad or do not otherwise fulfill the information need 

• Asking questions of a “professor” that help disambiguate a vague research assignment 

• Conducting effective preliminary information searches to help frame a research statement 

Access: Collect and/or retrieve information in digital environments. Information sources might be Web pages, databases, 
discussion groups, e-mail, or online descriptions of print media. Tasks include 

• Generating and combining search terms (key words) to satisfy the requirements of a particular research task 

• Efficiently browsing one or more resources to locate pertinent information 

• Deciding what types of resources might yield the most useful information for a particular need 

Evaluate: Judge whether information satisfies an information problem by determining authority, bias, timeliness, relevance, and 
other aspects of materials. Tasks include 

• Judging the relative usefulness of provided Web pages and online journal articles 

• Evaluating whether a database contains appropriately current and pertinent information 

• Deciding the extent to which a collection of resources sufficiently covers a research area 

Manage: Organize information to help you or others find it later, such as by 

• Categorizing e-mails into appropriate folders based on a critical view of the e-mails’ contents 

• Arranging personnel information into an organizational chart 

• Sorting files, e-mails, or database returns to clarify clusters of related information 

Integrate: Interpret and represent information, such as by using digital tools to synthesize, summarize, compare, and contrast 
information from multiple sources while 

• Comparing advertisements, e-mails, or Web sites from competing vendors by summarizing information into a table 

• Summarizing and synthesizing information from a variety of types of sources according to specific criteria in order 
to compare information and make a decision 

• Re-representing results from an academic or sports tournament into a spreadsheet to clarify standings and decide the 
need for playoffs 

Create: Adapt, apply, design, or construct information in digital environments, such as by 

• Editing and formatting a document according to a set of editorial specifications 

• Creating a presentation slide to support a position on a controversial topic 

• Creating a data display to clarify the relationship between academic and economic variables 

Communicate: Disseminate information tailored to a particular audience in an effective digital format, such as by 

• Formatting a document to make it more useful to a particular group 

• Transforming an e-mail into a succinct presentation to meet an audience’s needs 

• Selecting and organizing slides for distinct presentations to different audiences. 

• Designing a flyer to advertise to a distinct group of users 

Note. From “Testing Information Literacy in Digital Environments: The ETS iSkills™ Assessment” by I. R. Katz, 

2007, Information Technology and Libraries, 26(3), pp. 3-12. Copyright 2007 by Educational Testing Service (ETS). 
Used with permission of the ETS and the author. 
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Forum for Information Literacy, and the American Association of School Librarians. Panelists 
were selected to represent experience with diverse populations of students defined by 
race/ethnicity, geographic location, and institution type (high school, community or technical 
college, 4-year college or university). 

For the current study, the panelists were divided into two subpanels of nine members each 
using a stratified random sampling approach. Each subpanel had comparable representation from 
the three institution types as well as reflecting similar geographic and ethnic diversity. 

Table 2 shows characteristics of the panelists in each of the two subpanels. One subpanel 
(Panel A) worked on the Core iSkills assessment first, followed by the Advanced i Skills assessment. 
The other subpanel (Panel B) worked on the Advanced iSkills assessment first followed by the Core 
iSkills assessment. On the final day of the study, the two groups reconciled any differences in the cut 
score determinations for both tests. This counterbalanced design was used to mitigate the influence 
of the order in which the assessments are considered by the panelists. 

Table 2 



Characteristics of Panelists 





Panel A 


Panel B 




(Core - Advanced iSkills 


(Advanced - Core iSkills 




assessments) 


assessments) 




n = 9 


n = 9 


Gender 


Female 


5 


5 


Male 


4 


4 


Race/ethnicity 


Asian American 


1 




African American 


2 


5 


Hispanic 


1 




White (non-Hispanic) 


5 


4 


Institution type 


High school 


2 


3 


Community/technical college 


3 


3 


College or university 


4 


3 


Institution location 


Northeast 


2 


2 


Midwest 


2 


2 


South 


2 


1 


Southwest 


2 


1 


West 


1 


3 
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Process and Method 



Prior to the Meeting 

To fa mi liarize themselves with the iSkills assessments before the standard- setting 
meeting, all panelists completed both levels of the iSkills assessment. Panelists administered the 
tests to themselves, on their own computers, and experienced all steps of the test including 
background questions, the brief tutorial, and all tasks. Due to logistical constraints, panelists 
were not sent score reports and so received no feedback on their performance. 

In addition to completing the assessments, panelists reviewed the foundational level 
descriptions prior to the meeting. They were asked to compose two to three performance 
indicators — brief descriptions of observable student behaviors and outcomes — for each of the 
seven content areas of the foundational levels. The indicators were intended to clarify the 
meaning of each content area for the panelists. 

Panelists were sent these assignments in an e-mail approximately 2 weeks before the 
standard- setting study (Appendix B). All panelists acknowledged receiving the e-mail 
assignments. 

During the Meeting 

The agenda for the standard- setting study is included in Appendix C. The standard-setting 
study began with a general introduction to the session. Dr. Irvin Katz, senior research scientist for 
the iSkills program at Educational Testing Service (ETS), welcomed the panelists and introduced 
the ETS team responsible for the design and facilitation of the standard- setting process. Dr. 

Richard Tannenbaum from ETS’s Validity Research Center provided overviews of standard setting 
and the procedures to be followed by the panelists. Dr. Katz then led a discussion of the iSkills 
assessments, informed by the panelists’ experience with having completed the assessments on their 
own before the standard-setting meeting. This discussion was meant to reinforce the panelists’ 
familiarity with both assessments’ content as well as the purpose and population associated with 
each assessment. Panelists were encouraged to discuss, for example, what the test was covering 
(the major content areas), whether they thought any tasks would be particularly challenging for 
students, and whether there were any tasks that they thought addressed knowledge or skills that 
were particularly important for students to know or have. The panelists also had the opportunity to 
raise and discuss concerns about the tasks, although it was made clear that the primary purpose of 
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this test familiarization activity was not to conduct a review, per se. To maintain focus on the 
standard- setting process, but not to discount the panelists’ reactions to tasks, they were encouraged 
to write down specific concerns, which would be shared after the meeting with the ETS 
Assessment Development group. 

Panelists then reviewed both foundational level descriptions as developed by the policy 
council, using the descriptions to define the just qualified candidate (JQC). The JQC represents 
those students who have just enough ICT literacy skills to meet the challenges of entry-level 
college work (core) or upper-division college work and/or entry into the workplace 
(intermediate). The JQC is not the most able student, nor even the average performing student, 
but the student with just enough knowledge and skills to be considered meeting either the core 
foundational ICT literacy level (Core iSkills) or intermediate foundational ICT literacy level 
(Advanced iSkills). JQCs serve as the frame of reference against which the panelists judge the 
difficulty of each test item as part of the standard-setting process. 

For this review, the panel was divided into two groups that initially worked in separate 
rooms, with approximately equal numbers of panelists from Panels A and B working in each 
room. One room reviewed the core foundational level and the other reviewed the intermediate 
foundational level (applicable to the Advanced iSkills assessment). The current definitions 
describe what is expected of a student who just meets the core and intermediate foundational 
levels. When they originally defined the foundational levels, the policy council members were 
asked to focus on the JQC from the outset, rather than trying to define the full range of ICT 
literacy skills for each level. The discussions of the standard- setting panelists focused on their 
understanding of these descriptions and their fleshing out of the descriptions. Care was taken so 
that in refining terms, panelists did not alter the policy council’s definitions of the levels. The 
goal was to generate several knowledge and skill statements (performance indicators) that would 
adequately describe the JQC so that the panelists would have a clear picture of whom they 
considered to be just qualified. The intention was not to produce an exhaustive description. 
Drawing upon their own professional experience and their experience with having taken the 
assessments, each group produced three to five additional bullet points for each content area of 
the foundational levels that described the types of student behaviors or products that characterize 
students just meeting the foundational level. Because the descriptions were for the panel’s own 
use, panelists were permitted to use short-hand notation and brief bullets, if desired, rather than 
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extended prose. The focus was on ensuring that the descriptions were sufficient for the panelists 
to have a common understanding of the knowledge and skills expected of JQCs. 

After working separately, the full panel reconvened and each group presented its 
explication of the foundational level descriptions. Some modifications and clarifications to these 
bullet points were made during the full group discussion; the final expanded foundational level 
descriptions (core and intermediate) were handed out to the entire group for their reference 
during the standard-setting process. 

Next, Dr. Tannenbaum gave an introduction to standard setting and explained the 
standard- setting process and the specific steps the panelists would follow to make their standard- 
setting judgments. An example of how a cut score is computed was also provided. The panelists 
then had an opportunity to practice making their standard- setting judgments. Panelists were 
trained to complete a modified, extended Angoff approach (Hambleton & Plake, 1995). In brief, 
panelists were asked to consider the knowledge and skills being measured by an item and then to 
consider the average score that 100 JQCs — as articulated by the full panel — would be expected 
to receive on an item. As noted earlier, most items on the iSkills assessment are scored as correct 
(1), partially correct (0.5), or incorrect (0). (A small number of items are scored 1 or 0.) Panelists 
were to record their judgments (average scores) on scannable forms using the following scale: 0, 
0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. The easier an item is believed to be for JQCs, the 
higher the average score rating (a score closer to 1, or full credit); the more difficult the item is 
believed to be for JQCs, the lower the average score rating (a score closer to 0, or no credit). 
Panelists were encouraged to consider the following rule of thumb: Relatively easy items were in 
the 0.7 to 1 range; relatively difficult items were in the 0 to 0.3 range; and moderately 
easy/moderately difficult items were in the 0.4 to 0.6 range. Panelists also were reminded that 
any one test taker either scores 0, 0.5, or 1 on an item (0 or 1 on some items), but that they were 
to consider how, on average, 100 test takers would perform. 

The panelists then were given an opportunity to practice making their standard-setting 
judgments on one task that contained two items. The judgments were collected, summarized 
using an ETS-developed analysis program, and discussed. The feedback included the number of 
points each panelist believed a JQC would earn, the panel’s mean judgments and the standard 
deviation of judgments across panelists. The judgments were also summarized in terms of 
difficulty ranges as defined above (easy: 0 to 0.3, moderate: 0.4 to 0.6, difficult: 0.7 to 1). This 
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clustering was a useful way of showing the similarity of the judgments in relation to the overall 
judged difficulty of each item; that is, whether the majority of panelists (e.g., 67% or more) saw 
the same item as being in the same range of difficulty or if the panelists’ judgments were more 
spread out across the three ranges of difficulty. The overall cut score for the practice task was 
computed as it would be for the operational iSkills assessment — the sum of the points for each 
item. Each panelist’s overall cut score was displayed, as was the panel’s mean and standard 
deviation. The panelists were asked to share their judgment rationales so that others could 
understand their perspectives. The discussion focused on the connection between the knowledge 
and skills measured by an item and the knowledge and skills of JQCs. 

After practice, panelists completed an initial evaluation form (see Appendix G). The form 
asked the panelists to indicate the extent to which they understood the purpose of the standard- 
setting study, the extent to which explanations were clear, and whether or not they were ready to 
proceed with their actual standard- setting judgments. 

Once Dr. Tannenbaum verified that all panelists were ready to proceed, the standard- 
setting process began. From this point forward, the two subpanels (Panel A and B) worked 
completely independently from one another until they met together on the last day. Dr. 
Tannenbaum facilitated Panel A, which worked on first the Core iSkills assessment and then the 
Advanced iSkills assessment; Dr. Katz facilitated Panel B, which worked on first the Advanced 
iSkills assessment and then the Core iSkills assessment. Each subpanel went through three 
rounds of judgments, with feedback and discussion between each round. The following 
procedure was identical for Panels A and B. 

Round 1. Judgments were made on each item within a task (each task includes two to 
seven items) before moving to the next task. For each task, the ETS facilitator first provided an 
overview of the task and its scoring, including screen snapshots of full-credit responses and 
descriptions of the scoring rubrics. Each panelist then made the following judgment for each 
item within the task: Of 100 students that just meet the foundational level description, what 
would be the average score they would most likely earn? Panelists were asked to choose scores 
between 0 and 1, in increments of lOths; for example, 0.00, 0.10, 0.20 . . . 1.00. (Although a 
group of test takers, on average, may earn any score between 0 and 1, for the current purpose of 
recommending a single cut score for each assessment, lOth-point increments permit sufficient 
judgment variability, and choosing from a given set of scores reduces the cognitive load on the 
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panelists.) Panelists were reminded to consider the three difficulty ranges discussed during their 
training. Panelists worked independently in making these judgments, filling out a scannable 
score sheet. After panelists completed their judgments for all items within a task, the next task 
was introduced. After the first few tasks were completed, the standard- setting facilitator 
confirmed that the panelists understood the process and had no additional questions. The 
panelists then continued, completing their judgments for the remaining tasks (items within tasks). 
All score sheets were scanned and the data uploaded into a spreadsheet. 

Round 1 discussion. The spreadsheet served as the focal point for the discussions. The 
spreadsheet summarized the cut scores constructed from each panelist’s judgments as well as the 
panel’s mean and standard deviation of judgments. In addition, the spreadsheet highlighted any 
items for which there were relatively large discrepancies among panelists’ judgments. 
Highlighted items included those for which fewer than six of the nine panelists placed the item in 
the same difficulty range (0 to .3, .4 to .6, and .7 to 1). Primary discussion focused on the items 
for which there was not a majority consensus, but the members were encouraged to seek 
clarification on items for which there was consensus, if doing so would be helpful to them. All 
discussion centered on the connection between the knowledge and skills measured by the item 
and the knowledge and skills of JQCs. The goal of the discussion was to share the different 
perspectives of the members, not to argue for a single perspective. 

Discussions during Round 1 also included inspection of item difficulty data. The 
facilitators encouraged panelists not to view the difficulty data as the correct answer, but rather 
as a general indication of the difficulty of an item for the students taking the test. When judging 
the Core iSkills assessment, item difficulty data were provided from two groups: (a) first year 
students at a community college or high school seniors planning to attend a community college 
(. N = 340) and (b) first year students at a 4-year college or high school seniors planning to attend 
a 4-year college (. N = 2,302). These two groups were included because they perform somewhat 
differently on the Core iSkills assessment. Students entering or planning to enter a 4-year 
program tend to score higher on average than students entering or planning to enter a 2-year 
program (average scores of 157 and 141, respectively). The full group of students who took the 
Core iSkills assessment included students at different class levels. However, the core 
foundational level was designed to be descriptive of expectations for students just entering 
college. Thus, these two groups represent a meaningful subset of the data for consideration by 
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the standard-setting panel. When judging the Advanced iSkills assessment, item difficulty data 
were provided from the students targeted in the intermediate foundational level: juniors and 
seniors attending 4-year colleges (N = 2,068). 

Round 2 judgments, Round 2 discussions, and Round 3 judgments. The procedure for 
Round 1 was followed for the next round. During the judgments for Round 2, panelists indicated 
only changes to their previous item-level judgments. If a panelist heard something during the 
Round 1 discussion that changed his or her interpretation of item difficulty for JQCs, he or she 
was able to change his or her judgments in the second round. However, panelists were informed 
that they were not obligated or expected to change their Round 1 judgments. 

Following these judgments, panelists discussed the updated results, which included the 
same information provided for the Round 1 discussions. In addition, panelists were presented with 
the percentage of test takers meeting or exceeding the Round 2 computed cut score. For the Core 
iSkills assessment, this classification information was presented for the two applicable groups of 
test takers: (a) first year students at a community college or high school seniors planning to attend 
a community college and (b) first year students at a 4-year college or high school seniors planning 
to attend a 4-year college. For the Advanced iSkills assessment, this classification information was 
presented for upper-division (third and fourth year) students. Panelists then had an opportunity to 
change their judgments in a third round. At the completion of Round 3, each subpanel had its 
preliminary cut score for the assessment under consideration (Core for Panel A and Advanced for 
Panel B). Each subpanel then repeated the three-round standard- setting process for the other iSkills 
assessment (Advanced for Panel A and Core for Panel B). 

Full panel discussion. Panels A and B met to reconcile any differences between their 
respective cut scores for the Core and Advanced iSkills assessments. At the beginning of this 
round, panelists were shown the implications of their cut-score decisions based on assessment 
data: the percentage of students in the three comparison groups (two for Core and one for 
Advanced) that met or exceeded the selected cut scores. These classification outcomes were 
presented for each subpanel’s independent cut scores and for the average of the two subpanels. The 
combined panel considered these outcomes to inform their final recommendations of cut scores. 

The panelists then completed an evaluation form that asked for their overall reactions to the 
study, the factors that most influenced their judgments, and their comfort with the recommended cut 
scores (see Appendix H). 



10 




Results 



Initial Evaluations 

The panelists completed an initial evaluation form after practicing their standard- setting 
judgments. The evaluation form asked them to indicate the extent to which they understood the 
purpose of the standard-setting study, the extent to which explanations were clear, and whether 
or not they were ready to proceed with their actual standard- setting judgments. Table 3 presents 
the results of the initial evaluation. 

All 18 panelists responded to each evaluation question. All panelists either agreed or 
strongly agreed that they understood the purpose of the study, that things were explained clearly, 
that the training they received adequately prepared them, and that they were ready to proceed. 



Table 3 

Initial Evaluations 





Strongly 

agree 


Agree 


Strongly No 
Disagree disagree response 


I understand the purpose of the study 


16 


2 


0 


0 


0 


The facilitators explained things clearly during the 
large-group session 


16 


2 


0 


0 


0 


I understand the definitions of the foundational levels 


12 


6 


0 


0 


0 


The training in the standard-setting process adequately 
prepared me to make my standard- setting judgments 


11 


7 


0 


0 


0 


The explanation of how the recommended cut scores 
are computed was clear 


14 


4 


0 


0 


0 


The opportunity to practice helped clarify the standard- 
setting task for me 


13 


5 


0 


0 


0 


I understand how to make the standard- setting 
judgments 


15 


3 


0 


0 


0 


I am ready to proceed 


Yes 




No 






18 






0 
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Summary of Standard- Setting Judgments by Round 

Tables 4 through 7 present a summary of each round of standard- setting judgments for 
each subpanel’s judgments of each assessment. The numbers in each table reflect the 
recommended cut score — the number of raw points needed to meet or exceed a foundational 
level — of each panelist for each of the three rounds. Each subpanel’s (Panel A and Panel B) 
recommended cut score is reported, as are the standard deviation (SD) of panelists’ cut scores 
and the standard error of judgment (SEJ). 1 The SEJ is one way of estimating the reliability of the 
judgments. It indicates how close the mean cut score would likely be to the current mean cut 
score for other panels of educators similar in composition and experience to the current panel 
and similarly trained in the same standard-setting methods. A comparable panel’s cut score 
would be within 1 SEJ of the current mean cut score 68% of the time and within 2 SEJs 96% of 
the time. 



Table 4 



Panel A: Core Foundational Level Cut Scores by Round 



Core 


Round 1 


Round 2 


Round 3 


Panelist 01 


42.4 


44.4 


44.2 


Panelist 02 


35.8 


36.3 


36.2 


Panelist 03 


37.7 


37.9 


37.9 


Panelist 04 


36.0 


36.3 


36.3 


Panelist 05 


47.9 


46.0 


44.7 


Panelist 06 


35.5 


38.2 


37.9 


Panelist 07 


27.4 


28.4 


28.4 


Panelist 08 


49.5 


49.5 


38.2 


Panelist 09 


48.5 


48.3 


48.3 


Mean cut 


40.1 


40.6 


39.1 


SD 


7.5 


6.9 


5.9 


SEJ 


2.5 


2.3 


2.0 



Note. The SD indicates the amount of variability among the panelists’ cut score recommendations. 
The smaller the SD, the greater the consistency in the individual recommendations. SD = standard 
deviation, SEJ = standard error of judgment. 
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Table 5 



Panel B: Core Foundational Level Cut Scores by Round 



Core 


Round 1 


Round 2 


Round 3 


Panelist 10 


47.2 


47.2 


47.2 


Panelist 11 


42.9 


43.2 


43.4 


Panelist 12 


52.1 


51.8 


51.8 


Panelist 13 


45.0 


45.5 


45.5 


Panelist 14 


43.8 


44.4 


44.1 


Panelist 15 


44.1 


44.6 


44.6 


Panelist 16 


43.1 


43.1 


43.1 


Panelist 17 


37.2 


36.7 


36.8 


Panelist 18 


42.5 


43.0 


43.0 


Mean cut 


44.2 


44.4 


44.4 


SD 


4.0 


4.0 


4.0 


SEJ 


0.75 


0.75 


0.75 


Note. SD = 


standard deviation, SEJ = 


standard error of judgment. 





Table 6 

Panel A: Intermediate Foundational Level Cut Scores by Round 


Intermediate 


Round 1 


Round 2 


Round 3 


Panelist 01 


39.9 


39.6 


39.4 


Panelist 02 


38.5 


35.1 


35.1 


Panelist 03 


33.6 


33.6 


33.6 


Panelist 04 


41.3 


41.5 


41.7 


Panelist 05 


45.3 


41.5 


40.4 


Panelist 06 


35.2 


34.7 


34.0 


Panelist 07 


35.1 


35.7 


35.7 


Panelist 08 


43.0 


42.9 


39.8 


Panelist 09 


50.2 


49.5 


49.5 


Mean cut 


40.2 


39.3 


38.8 


SD 


5.4 


5.1 


5.0 


SEJ 


1.8 


1.7 


1.7 


Note. SD = standard deviation, SEJ = 


standard error of judgment. 
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Table 7 



Panel B: Intermediate Foundational Level Cut Scores by Round 



Intermediate 


Round 1 


Round 2 


Round 3 


Panelist 10 


38.3 


38.7 


38.7 


Panelist 11 


25.7 


35.5 


36.0 


Panelist 12 


42.0 


42.3 


42.3 


Panelist 13 


32.3 


37.3 


37.3 


Panelist 14 


33.1 


37.4 


37.8 


Panelist 15 


38.9 


39.9 


40.3 


Panelist 16 


42.1 


41.9 


41.9 


Panelist 17 


29.8 


35.9 


36.3 


Panelist 18 


39.6 


40.1 


40.3 


Mean cut 


35.8 


38.8 


39.0 


SD 


5.8 


2.5 


2.3 


SEJ 


1.9 


0.83 


0.77 



Note. SD = standard deviation, SEJ = standard error of judgment. 



In most standard-setting studies, Round 1 judgments are made without discussion among 
the panelists. The most variability in judgments, therefore, is typically present in the first round. 
Round 2 and Round 3 judgments, however, are informed by committee discussion; because of 
this, it is common to see a decrease both in the SD and SEJs. This decrease was observed in most 
cases for both panels on each of the two assessments. 

The mean of the Round 3 judgments represents each subpanel’s final recommended cut 
score for each assessment. 

Recommended Cut Scores by Assessment 

Each cut score accepted by the full panel (Panels A and B combined) was the mean 
of the cut scores of the two subpanels. The full panel recommended that the core 
foundational level of ICT literacy skill correspond to a score of 165 (41.8 raw score) on the 
Core iSkills assessment. Considering the first year students who have taken the Core i Skills 
assessment since 2006, this cut score results in approximately 24% of community/technical 
college first year students (and appropriate high school seniors) and 39% of 4-year college 
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freshmen (and appropriate high school seniors) meeting or exceeding the core foundational 
level of ICT literacy skill. The full panel recommended that the intermediate foundational 
level of ICT literacy skill corresponds to a score of 575 (39.0 raw score) on the Advanced 
iSkills assessment. Considering the juniors and seniors who have taken the Advanced 
iSkills assessment since 2006, this cut score results in approximately 27% of these students 
meeting or exceeding the intermediate foundational level of ICT literacy skill. 

Table 8 shows additional data on the cut-score recommendations, including the 
Round 3 cut-score recommendation from each separate panel, the combined-panel mean cut 
score, and the range of cut scores that fall within 2 SEJs of the mean cut score. ETS suggested 
that the policy council stay within this range when selecting a final cut score for each 
assessment to maintain some alignment with the panel’s recommendations. In addition to these 
data, Table 8 shows the scale score corresponding to the cut scores as well as the percentage of 
students who would be classified as meeting or exceeding the cut score. Higher cut scores 
result in fewer students being classified as meeting a foundational level. 



Table 8 

Data on Cut Score Recommendations 







Panel A 
(Core - 
Advanced 
iSkills 

assessments) 


Panel B 
(Advanced - 
Core 
iSkills 

assessments) 


Combined-p anel 
recommendation 


Range of 
acceptable 
cut scores 
(+/- 2 SEJ) 


Core 


Raw score (SD) 


39.1 (5.9) 


44.4 (4.0) 


41.8 (5.6) 


39.2 - 44.4 


foundational 

level 


Scale score 


160 


175 


165 


160-175 


% classified 
(CC 1 st years) 


29% 


16% 


24% 


29 - 16% 




% classified 

(4-year 

freshmen) 


46% 


26% 


39% 


46 - 26% 


Intermediate 


Raw score (SD) 


38.8 (5.0) 


39 (2.3) 


38.9(3.8) 


37.1 -40.7 


foundational 

level 


Scale score 


575 


575 


575 


570-580 


% classified 
(juniors & 
seniors) 


27% 


27% 


27% 


34 - 24% 



Note. SD = standard deviation, SEJ = standard error of judgment. 
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While the recommendation of Panel A on the core foundational level (39.1) appears 
to fall outside the range of acceptable cut scores (39.2-44.4), this discrepancy has no 
practical import as raw scores on the assessment are assigned in 0.5 increments: 39.1 and 
39.2 correspond to the same scale score (160). 

Figures 1 through 3 present histograms of scores for the three comparison groups of 
students described earlier: (a) first year students at community colleges and high school 
seniors that plan to attend a community college, (b) first year students at 4-year colleges and 
high school seniors that plan to attend a 4-year college, and (c) juniors and seniors at 4-year 
colleges. The impact of the recommended cut scores for these three groups are indicated by 
the colored bars: Gray bars indicate those students meeting or exceeding the recommended 
cut scores while white bars indicate students falling below the cut score. Note that shifting 
the recommended cut scores will change the proportion of students who meet or exceed a 
foundational level. 



cutscoregroups: Core 2yr 



50- 



40- 




40 60 80 100 120 140 160 180 200 220 



Mean = 140.9 
Std. Dev. = 33.864 
N = 340 



SCALESCORE 



Figure 1. Histogram of scores from first year students at community or technical colleges 
on the Core iSkills assessment. 

Note. Gray bars are at or above the consensus cut score of 165; white bars are below the cut score. 
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cutscoregroups: Core 4yr 




Mean = 156.99 
Std. Dev. = 28.208 
N = 2,302 



SCALESCORE 



Figure 2. Histogram of scores from freshmen at 4-year colleges or universities on the Core 
iSkills assessment. 

Note. Gray bars are at or above the consensus cut score of 165; white bars are below the cut score. 



cutscoregroups; Advanced 




Mean = 556.15 
Std. Dev. = 32.28 
N = 2,068 



SCALESCORE 

Figure 3. Histogram of scores from juniors and seniors at 4-year colleges or universities on 
the Advanced iSkills assessment. 



Note. Gray bars are above the consensus cut score of 575; white bars are below the cut score. 
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Final Evaluations 

Panelists completed an evaluation form at the conclusion of the standard- setting study. 

The evaluation form asked panelists to confirm some of their reactions from the initial evaluation 
form, but also asked for feedback regarding the standard- setting process, what influenced their 
judgments, and their comfort with the panel’s Round 3 (final) recommended cut score. Tables 9 
through 1 1 present the results of the final evaluations. 

All 18 panelists responded to each question. All panelists confirmed that they understood the 
purpose of the study, that things were explained clearly, and that the training they received 
adequately prepared them. All panelists indicated that they understood how the cut score was 
calculated and that the opportunity for feedback and discussion between rounds was helpful. 

The majority of panelists reported that the definition of the JQC, their own professional 
experience, and the item-level data most influenced their judgments. All (100%) thought that the 
cut score reached for the Advanced iSkills assessment was about right; 17 of the 18 panelists 
(94%) thought the cut score reached for the Core iSkills assessment was about right. 



Table 9 



Final Evaluation: Overall Opinions 





Strongly 

agree 


Agree 


Disagree 


Strongly 

disagree 


No 

response 


I understood the purpose of the study. 


18 


0 


0 


0 


0 


The instructions and explanations provided 
by the room facilitator were clear. 


17 


1 


0 


0 


0 


The training in the standard- setting process 
adequately prepared me to make my 
standard- setting judgments. 


11 


7 


0 


0 


0 


The opportunity for feedback and discussion 
between rounds was helpful. 


15 


3 


0 


0 


0 


The inclusion of the item data was helpful. 


17 


1 


0 


0 


0 


The inclusion of the classification 


14 


a 


1 


0 


0 


percentages was helpful. 


J 


The opportunity to reach consensus during 
the last day was helpful. 


15 


3 


0 


0 


0 


The overall process of making the standard- 
setting judgments was easy to follow. 


11 


7 


0 


0 


0 
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Table 10 



Final Evaluation: Influence of Materials and Data 



How influential was each of the following factors 


Very 


Somewhat 


Not 


No 


in guiding your standard-setting judgments? 


influential 


influential 


influential 


response 


The definition of the. foundation levels 


16 


2 


0 


0 


The between-round feedback and discussions 


10 


7 


1 


0 


The standard- setting judgments of other panelists 


4 


12 


2 


0 


The item-level data 


13 


4 


1 


0 


The classification percentages 


8 


7 


3 


0 


My own professional experience 


14 


3 


1 


0 


Table 11 










Final Evaluation: Agreement With Recommended Cut Scores 








Too low 


About right 


Too high 


No 

response 


The Round 3 (final) recommended cut score for 
the Core assessment is: 


0 


17 


1 


0 


The Round 3 (final) recommended cut score for 
the Advanced assessment is: 


0 


18 


0 


0 



Conclusions 

This document reports a standard- setting study for the Core and Advanced iSkills 
assessments. The 18 panelists reported understanding the standard-setting process. Panelists also 
agreed with the consensus cut scores representing the mean of the two panels. The panel 
recommends that a cut score of 165 on the Core iSkills assessment corresponds to the core 
foundational level as set forth by the National ICT Literacy Policy Council. On the Advanced 
iSkills assessment, the panel recommends that a cut score of 575 corresponds to the intermediate 
foundation level. Because these recommendations were reached through the converging 
recommendations of independent subpanels, the National ICT Literacy Policy Council should 
feel confident that these cut scores reflect the defined foundational levels. 



19 




Postscript 

On October 11, 2007, the authors presented this report and recommendations to the 
National ICT Literacy Policy Council. In December 2007, after reviewing the report, the policy 
council approved the recommended cut scores. 
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Notes 

1 The SEJ assumes that panelists were selected at random from a population of panelists. In most 
instances, including the current study, this is not likely to be true. Therefore, the SEJ should 
be interpreted as an approximation of each cut score’s replicability. 
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Appendix B 

Premeeting Assignments 



Colleagues: 

The July 20-23 standard- setting (cut score) panel for the iSkills™ Core and Advanced 
assessments is fast approaching. We look forward to meeting and working with you on this 
important project; please see the attached draft agenda for an overview of the meeting. During 
the study we will provide you with the background information and training necessary to 
complete the standard-setting process. However, our experience shows that some pre-meeting 
preparation greatly facilitates the standard setting, making the process simpler and more 
rewarding for all participants. To that end, we ask that you complete the following “assignments” 
before you travel to the meeting. 

Please complete the following two assignments before July 17 th : 

1. Take both the Core iSkills assessment and the Advanced iSkdls assessment. Each assessment 
can be accessed via the attached instructions; please be sure to take both assessments. It is 
critical to the process that you become well-acquainted with the details of each assessment as 
well as the relative difficulty of the Core and Advanced assessments. The best way to achieve 
this goal is for you to go through the assessment as any test-taker would. The more fa mi liar 
you are with each assessment, the easier it will be for you to judge the difficulty of the items 
for students, which is the focus of the standard-setting process. How well you do on the 
assessment will NOT be shared; this activity is strictly for you to better understand each 
assessment. If there are any issues you want to raise about particular tasks at the meeting, 
please be sure to note the assessment name (Core or Advanced) as well as some 
distinguishing content from the tasks (e.g., “Vietnam project,” “Great Garloo”). 

Some of you might have already reviewed one or both of the iSkills assessments at some 
point in the past. However, if you have not taken both assessments for yourself in the past 
month, please use this opportunity to reacquaint yourself with the assessments. 



25 




2. Please review the two foundational level descriptions (attached). The foundational levels 
were developed by the National Policy Council and informed by the ACRL Standards. These 
descriptions articulate what ICT literacy skills students should have if they are meeting 
expectations. For the Core assessment, the foundational level describes what is expected of 
students who are leaving high school and entering their first year of college, either the first 
year of community college or the first year of a 4-year college/university program. For the 
Advanced iSkills assessment, the foundational level describes what is expected of students 
who either are entering then - junior year in a 4-year program or entering the workforce. 

As you review each level, please jot down one or two performance indicators for each of the 
seven skill areas in the definitions . Ask yourself, “what specifically would I expect to see 
(behavior or product) from a student who has the skill as described; what can she or he do?” 
We will continue this exercise during the meeting to help flesh out each foundational level. 
When you have finished reviewing the foundational levels, please write down your estimate 
of the percentage of the students you teach and/or interact with who meet or exceed the 
Foundation Level for (a) the Core assessment population (high school seniors and college 
freshmen) and (b) the Advanced assessment population (students moving to their junior year 
of college). Please bring your notes to the July meeting. 

Also, if you have the time, please read the attached background paper on the iSkills assessment. 

This paper will provide you with an understanding of the purpose and goals of the assessments, 

and an understanding of how they were developed. 

We do realize that your time is valuable and finite. But please do try to complete the first two 

assignments; and we encourage you — time permitting — to read the background paper. 

Finally, please acknowledge receipt of this e-mail. 

Again, we look forward to working with you ! 

Regards, 

Irv Katz Rick Tannenbaum 
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Appendix C 
Meeting Agenda 



July 20 - 23 
Agenda 

July 20, 2007 

8:30-8:45 AM Introduction and welcome 



8:45 -9:05 Orientation to standard setting and workshop events 
9:05 - 9:30 Discuss iSkills assessments 



9:30 - 1 1:00 Flesh out foundational level descriptions 



11 : 00 - 12:00 
12:00- 1:00 PM 
1:00-5:00 
5:00 



Standard setting training and practice 
Lunch 

Round 1 judgments (complete 50% to 75% of the items) 
Adjourn (Day 1) 



July 21, 2007 

8:30- 10:30 AM 

10:30-10:45 

10:45 - 12:00 

12:00- 1:00 PM 

1:00-1:45 

1:45-2:30 

2:30-2:45 

2:45- 4:00 

4:00- 4:45 

4:45 



Brief recap of previous day. Complete Round 1 judgments 

Data entry; break 

Discuss Round 1 judgments 

Lunch 

Discuss Round 1 judgments 

Round 2 judgments (only for items you want to change) 

Data entry; break 

Discuss Round 2 judgments 

Round 3 judgments (only for items you want to change) 
Adjourn (Day 2); data entry 
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July 22, 2007 

8:30 - 9:00 AM Review foundational level description for second iSkills test; 

discuss differences from other foundational level description 

9:00 - 12:00 Round 1 judgments (for second iSkills test) 

12:00 - 1 :00 PM Lunch; data entry 

1:00 - 2:30 Discuss Round 1 judgments 

2:30-3:00 Round 2 judgments (only for items you want to change) 

3:00-3:15 Data entry, break 

3:15- 4:00 Discuss Round 2 judgments 

4:00 - 4:30 Round 3 judgments (only for items you want to change) 



4:30 Adjourn (Day 3); data entry 



July 23, 2007 

8:30- 1 1 :00 AM Consensus on final recommended cut scores; completion of final 
evaluation form 

1 1 :00 — 11:15 Wrap up and adjourn (end of workshop) 

Breaks at approximately 10:30 AM and 3:00 PM each day (except July 23) 
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Appendix D 

ETS Nondisclosure Agreement 



This Non-Disclosure Agreement (the “Agreement”) is effective as of the date written 
below between Educational Testing Service, located at Rosedale Rd., Princeton, NJ 08541 
(“ETS”) and the individual as listed below (the “Participant”). This Agreement is based on the 
understanding that both ETS and the Participant have mutually beneficial interests. This 
Agreement will facilitate discussions and dealings between ETS and the Participant regarding the 
iSkills Assessment Cut Score Panel Meeting (the “Purpose”). ETS and the Participant recognize 
that such discussions may involve the disclosure of confidential information and agree to protect 
this confidential information from unauthorized use and disclosure. In consideration of the 
disclosure of such information by ETS, the parties agree as follows: 

This Agreement will apply to any confidential and proprietary information disclosed by 
ETS including, any information concerning ETS’s product development plans, technology, 
software, intellectual property, marketing or customer information, financial information, trade 
secrets, business plans, any student, examinee or candidate testing or demographic data, and 
whether such information is in written, oral or visual form (collectively, “Confidential 
Information”). 

Participant agrees to hold ETS’s Confidential Information in confidence, not to disclose 
such Confidential Information to any individual or company (except as may be required within 
Participant’s institution or organization for the sole purpose of furthering the Purpose), and not to 
use any Confidential Information for any purpose except for the Purpose stated above. 
Confidential Information will not include information which: is generally known or available to 
the public; was lawfully known by the Participant prior to the effective date of this Agreement; 
or was acquired from a third party which has no obligation of confidentiality to ETS. 

If Participant or ETS decides not to proceed with the Purpose, it will promptly notify the 
other. Upon ETS’s request, the Participant will promptly return all tangible items received from 
ETS and all copies thereof, and will destroy any abstracts or summations of such information it 
has made. The Participant agree that nothing contained in this Agreement grants any rights to it, 
by license or otherwise, to any Confidential Information except as specified in this Agreement. 

This Agreement will be governed by the laws of the State of New Jersey (excluding its 
body of law controlling conflicts of law). The terms and conditions of this Agreement shall inure 
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to the benefit of and be binding upon the respective successors and assigns of the parties hereto. 
If any provision of this Agreement is or becomes illegal, or unenforceable, the remaining 
portions of the Agreement shall remain separately valid and in full force and effect. 

The Participant has executed this Agreement by signing below. 

The “Participant”: 

By: 

Signature 

Name, Title: 

Date: 
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Appendix E 

Panelist Biographical Data Form 

National ICT Literacy Cut Score Panel 
Panelist Biographical Data Form 

Your Name: 

Panelist’s Name: 

Panelist’s Address: 

Panelist’s City, State, Zip: 

Panelist’s Institution Name: 

1 . Does the panelist teach or in some other way work with students and have an 

opportunity to observe their ICT literacy skills? 

Yes No 

If “yes,” indicate the population with whom the panelist primarily interacts (e.g. 
high school seniors, college freshmen, employees, etc.) and the nature of his/her 
responsibilities: 



2 . 



Please summarize the panelist’s qualifications as an ICT Literacy expert and a 
person knowledgeable about student/employee ICT literacy abilities: 





3. Does the panelist work with other ICT-oriented teaching professionals and have a 
good working knowledge of the ICT Literacy skills needed to succeed in higher 
education and/or the workplace? 

Yes No 

4. How do you expect that the panelist describes himself/herself? (check all that apply) 

American Indian or Alaskan Native 

Asian, Asian American, or Pacific Islander 

Black or African American 

Mexican, Mexican American, or Chicano 

Puerto Rican 

Other Hispanic or Latin American 

Middle Eastern / North African 

White (non-Hispanic) 

Some other way not listed above (please specify): 

5. Panelist’s gender: 

Female Male 

6. Region of country where the panelist teaches or works: 

East Midwest Northwest 

West South Southwest 

7. Panelist’s institutional location: 

Urban Suburban Rural 

8. Panelist’s institutional type: 

High School Community/Technical College 

Four-year College or University Employer 

9. Number of students at panelist’s institution: 

10. If an Employer, what type of organization (e.g., service, technology, 

manufacturing, transportation, health, financial, . . . ): 

1 1. What is your recommendation on Cut Score panel assignment for this person: 

Core level. The Core level assesses readiness to meet the ICT literacy 

challenges of higher education. It is taken by high school seniors and college 
freshmen. 

Advanced level. The Advanced level assesses readiness to meet the ICT 

literacy challenges of upper-division coursework, including transitioning from a 
2-yr institution to a 4-yr institution or the workforce. It is taken by college 
sophomores or juniors. 
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Appendix F 

Foundational Level Descriptions 



Core Foundational ICT Literacy Skills 

(relevant to Core iSkills assessment) 

Demonstrate abilities to define tasks and needs, and to access and manage information in an 
effective, efficient, and ethical manner through the appropriate selection and application of 
information and communication technology to succeed in lower division (postsecondary) studies 
and/or the workplace. 

Define: Articulate a need for information as one or more relevant, focused, and manageable 
questions. Know where to locate and gain understanding of acceptable, common definitions of 
terms associated with the needed information. 

Access: Search, find, and retrieve information from a variety of print and electronic resources 
(e.g., databases, Internet). 

Evaluate: Judge the currency, appropriateness, and adequacy of information and information 
sources for a specific purpose. 

Manage: Conduct a rudimentary and preliminary organization of accessed information for 
retrieval and future application. 

Integrate: Extract and combine information from a variety of sources and draw fundamental 
conclusions. 

Create: Summarize and adapt information to describe an event, express an opinion, or support a 
basic argument, viewpoint, or position. 

Communicate: Adapt and present information for a peer audience. 
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Intermediate Foundational ICT Literacy Skills 

(relevant to Advanced iSkills assessment) 



Demonstrate abilities that build on the core foundational skills of ICT literacy (define, access, 
manage, and use information). The learner selects and applies appropriate ICT tools to 
synthesize, integrate, and assimilate information, to evaluate evidence and infer conclusions, to 
create and reflect on information processes and products, and to communicate results in a 
persuasive, ethical, and legal manner. These abilities are demonstrated at a skill level necessary 
to succeed in 3 rd year postsecondary studies and/or the workplace. 

Define: Articulate a need for information that defines a hypothesis or problem in operational 
terms. 

Access: Develop and apply a systematic strategy for ethically and legally finding, retrieving, and 
sorting information from a variety of relevant sources, representing a wide spectrum of 
perspectives, acknowledging sources appropriately. 

Evaluate: Judge veracity, bias, primacy, persuasiveness, and completeness of information and 
information sources for a specific purpose. 

Manage: Develop and apply a comprehensive system to classify and prioritize information in 
order to identify and clarify interrelationships. 

Integrate: Synthesize information from a variety of sources and perspectives, compare and 
contrast arguments, identify trends and patterns, and infer conclusions. 

Create: Generate information new to the learner through critical review and revision of 
assimilated information. Develop supported arguments and warranted conclusions to address the 
task at hand. 

Communicate: Communicate information persuasively to meet needs of various audiences 
through the use of an appropriate medium. 
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Appendix G 

Initial Evaluation/Ready-to-Proceed Form 



Panelist ID 

iSkills™ 

Initial Evaluation/Ready-to-Proceed Form 



Please read each statement and place an “X” in the box to represent vour response. 





Strongly 

Agree 


Agree 

•• 


Disagree 


Strongly 

Disagree 


• I understand the puipose of the study. 










* The facilitators explained things clearly during 
the large-group session. 










• I understand the definitions of the Foundational 
Levels. 










• The training in the standard setting process 
adequately prepared me to make my standard 
setting j udgments. 










• The explanation of how the recommended cut 
scores are computed was clear. 










• The opportunity to practice helped clarify the 
standard setting task for me. 










• 1 understand how to make the standard setting 
judgments. 











I arn ready to proceed and to make my standard setting judgments. 

Yes No 



If no , what other information/explanations do you need before making your standard setting judgments? 



Date 



Signature 
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Appendix H 
Final Evaluation Form 



Panelist ID 



iSkills™ 

Final Evaluation Form 



Please read each statement and place an “X” in the box to represent your response. 





Strongly 

Agree 


Agree 


Disagree 


Strongly 

Disagree 


• I understood the purpose of the study. 










• The instructions and explanations provided 
by my room facilitator were clear. 










• The training in the standard setting process 
adequately prepared me to make my standard 
setting judgments. 










• The opportunity for feedback and discussion 
between rounds was helpful. 










• The inclusion of the item data was helpful. 










• The inclusion of the classification 
percentages was helpful. 










• The opportunity to reach consensus during 
the last day was helpful. 










• The overall process of making the standard 
setting judgments was easy to follow. 












piSfSlspii 






.. nil 



How influential was each of the following factors in guiding your standard setting j udgments? 





Very 

Influential 


Somewhat 

Influential 


Not 

Influential 


• The definition of the Foundational levels 








• The between-round feedback and discussions 








• The standard setting judgments of other 
panelists 








• The item-level data 








• The classification percentages 








• My own professional experience 









The Round 3 (final) recommended cut score for the Core assessment is: (Check one.) 

□ Too Low □ About Right □ Too High 

The Round 3 (final) recommended cut score for the Advanced assessment is: (Check one.) 

□ Too Low □ About Right □ Too High 



If you would like to add comments, please use the back of this form . 
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Appendix I 
List of Panelists 



Ismail 


Abdullahi 


North Carolina Central University 


Durham, NC 


Penny 


Beile 


University of Central Florida 


Orlando, FL 


Diego 


Bonilla 


California State University, Sacramento 


Sacramento, CA 


Stephanie 


Brasley 


California State University Chancellor's 
Office 


Long Beach, CA 


Charles 


DeSassure 


Tarrant County Community College 


Arlington, TX 


Valerie 


Edwards 


Monona Grove High School 


Monona, WI 


Donald 


Forbes 


Guilford Technical Community College 


Jamestown, NC 


Mary Ann 


Harlan 


Areata High School 


Areata, CA 


Patricia 


Harris 


Oakland Community College 


Auburn Hills, MI 


Angela E. 


Hedley 


Health Careers Academy High School 


Boston, MA 


Gerald (Joe) 


Jemigan 


Tarrant County Community College 


Fort Worth, TX 


Yves 


Labissiere 


Portland State University 


Portland, OR 


LaRoi 


Lawton 


Bronx Community College 


Bronx, NY 


Emily M. 


Okada 


Indiana University 


Bloomington, IN 


Edward 


Owusu-Ansah 


Kingsborough Community College 


Brooklyn, NY 


Carrol 


Perrino 


Morgan State University 


Baltimore, MD 


Dawn P. 


Vaughn 


Cherry Creek High School 


Greenwood Village, CO 


Ann 


Vickman 


South Texas Independent School 
District - Biblioteca Las Americas 


Mercedes, TX 
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